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Abstract 

We report on a search for the flavor-changing neutral current decay D° — ► 
using 50 x 10 6 events recorded with a dimuon trigger in interactions of 920 GeV 
protons with nuclei by the HERA-B experiment. We find no evidence for such decays 
and set a 90% confidence level upper limit on the branching fraction £>(D° — > / u + / u _ ) < 
2.0 x 10~ 6 . 
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1 Introduction 



The decay D — > is sensitive to flavor-changing neutral currents 

(FCNC), which, due to the GIM mechanism [1], are forbidden at lowest order 
and strongly suppressed at second order in the Standard Model (SM). In the 
SM, the expected contribution of short distance processes to the branching 
fraction for this decay is of the order of 1CT 19 [2,3], well below the sensitiv- 
ity of current experiments. Long distance effects may enhance the branching 
fraction to roughly 1CT 13 [3,4], still undetectable by foreseeable experiments. 
However, several extensions of the SM including the Minimal Supersymmetric 
Standard Model (MSSM), models with multiple Higgs doublets, with hori- 
zontal gauge bosons, and with extra fermions, predict an enhancement of the 
branching fraction by several orders of magnitude. A comprehensive study of 
this issue has been presented in Burdman et al. [4] (see also references therein). 
According to this report, an MSSM variant with R-parity violation predicts 
the highest estimated branching fraction, 3.5 x 10~ 6 . Thus, the value of the 
branching fraction of the D° — > decay mode is exquisitely sensitive to 

physics beyond the SM. Recently, the CDF Collaboration has published a new 
upper limit of 2.5 x KT 6 at the 90% confidence level [5]. 

In this letter we report on a search for the D° — > fi + fi~ decay using 50 x 10 6 
events recorded with a dimuon trigger in interactions of 920 GeV protons with 
nuclei in the experiment HERA-B [6]. The data were recorded during the 
2002-2003 HERA running period. The branching fraction computation relies 
on normalizing the number of events in the D° signal region to the number 
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of reconstructed J/ip — > events [7] since possible biases arising from the 

dimuon trigger and muon identification largely cancel. Thus, for proton inter- 
actions on a single target of atomic weight A, the branching fraction limit can 
be determined using the following inequality: 



£(D C 



H+H ) < 



L^j/v, 



a D oe D o 



pA 
pA 



(1) 



where 

• n c ; is the upper limit on the number of D° — > decays; 

• Nj/^ is the number of observed J/tp — > events; 

• a D o and ojm are the acceptances for D° — > and J/^ — > fi + fi~ after 
event quality and particle identification cuts are applied (i.e., cuts applied 
to both channels); 

• e D o is the efficiency for D° — > of cuts designed to select secondary 
vertices (i.e., cuts not applied to J /if) — > /i + /i~); 

• and are the production cross sections per target nucleus for D° 
and J/^; 

• B(J/ip — > = (5.88 ±0.10)% [8] is the branching fraction for J/^> — > n + ji~ . 

The terms od", oj/^ and 6^0 are evaluated with a complete Monte Carlo sim- 
ulation. Equation (1) also requires knowledge of the relative production cross 
sections of D° and J/ip. Large errors on the available measurements of the D° 
cross section dominate the systematic error. 



2 The Detector and Trigger 



The HERA-B fixed-target spectrometer [6] operates at the 920 GeV proton 
beam of the HERA storage ring at DESY and features a vertex detector and 
extensive tracking and particle identification systems. It has a large geomet- 
rical coverage from 15 mrad to 220 mrad in the bending plane and 15 mrad 
to 160 mrad in the vertical plane. Figure 1 shows a plan view of the detector 
in the configuration of the 2002-2003 data run. 

The target system [9] consists of two stations of four wires each. The wires are 
positioned above, below, and on either side of the beam and are made from 
various materials including carbon, titanium, and tungsten. The stations are 
separated by 40 mm along the beam direction. The wires can be individually 
moved into the halo of the HERA proton beam and the interaction rate for 
each inserted wire can be adjusted independently. 

The Vertex Detector System (VDS) [10] is a forward microvertex detector 
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integrated into the HERA proton ring. It provides a precise measurement of 
primary and secondary vertices. The VDS consists of 7 stations (4 stereo views) 
of double-sided silicon strip detectors (50 x 70 mm, 50 /zm pitch) integrated 
into a Roman pot system inside a vacuum vessel and operated as near as 10 mm 
from the beam. An additional station is mounted immediately downstream of 
the 3 mm thick aluminum window of the vacuum vessel. 

The first station of the main tracker is placed upstream of the 2.13 T-m spec- 
trometer dipole magnet. The remaining 6 tracking stations extend from the 
downstream end of the magnet to the electromagnetic calorimeter (ECAL) at 
13 m downstream of the target. Each tracking station is divided into inner and 
outer detectors. The large area Outer Tracker (OTR) [11] consists of 95,000 
channels of honey-comb drift cells and covers the region starting from 200 mm 
from the beam center. The region starting from the beam pipe and extending 
up to the start of the OTR acceptance is covered by micro-strip gas chambers 
with GEM foils (Inner Tracker or ITR [12]). 

Particle identification is performed by a Ring Imaging Cherenkov detector 
(RICH) [13], the ECAL [14] and a muon detector (MUON) [15]. The MUON 
detector is segmented into four superlayers. Iron and concrete shielding ex- 
tends from just behind the ECAL to the penultimate MUON superlayer, ex- 
cept for gaps for the superlayers themselves. The first two superlayers consist 
of three layers of tube chambers with different stereo angles. The last two 
superlayers each consist of one layer of pad chambers. 

For the sample considered here, triggers are initiated by coincidences of pad 
chamber hits ( "pretriggers" ) from the third and last MUON superlayers [16]. 
Starting from the pretrigger coordinates, the First Level Trigger (FLT) [17] 
searches for tracks of charged particles in the MUON tracking layers and four 
of the main tracker stations (OTR only). The momentum of a candidate track 
is inferred by assuming the track originates at the target wires and calculating 
the bending angle within the magnetic field. Triggered events are required to 
have at least one FLT track and at least one additional pretrigger candidate. 

The Second Level Trigger (SLT) [18] processor farm receives the track and 
pretrigger candidates from the FLT. Starting from regions of interest defined 
by the pretriggers, it searches for tracks in the main tracker, extrapolates 
those found through the magnet and attempts to follow them through the 
VDS. Events with at least two fully reconstructed tracks which form a good 
vertex are triggered. Events passing the SLT are assembled and transferred to 
a fourth level farm (no third level algorithm is applied), where a fraction of the 
events is fully reconstructed online to provide monitoring. No event selection 
is applied at the fourth level. The final archiving rate is about 100 Hz. 
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3 Monte Carlo Simulation 



A Monte Carlo (MC) simulation is used to determine the D° — > jjL + ji~ and 
J/ip — > acceptance and efficiency terms in equation (1) and to estimate 

the background due to D° — > K~7r + and D° — > 7r + 7r~ decays. The Monte Carlo 
events for pA — ■> D°+X are generated in two steps. First, a cc pair is generated 
with PYTHIA 5.7 and hadronized with JETSET 7.4 [19] such that a D° is 
always produced. The fractional longitudinal momentum (xp) distribution of 
the D° is forced into the form (1 — |xp|) n and its transverse momentum (jpr) 
distribution is forced into the form exp(— bp^) by an appropriate selection 
of events. For n and b, we use averages of measurements by E653 [20] and 
E743 [21]: n = 7.7 ± 1.4, b = 0.83 ± 0.07 (GeV/cT 2 . After the generation 
of the D° is completed, the remaining energy is given as input to FRITIOF 
7.02 [22], which generates the underlying event taking into account further 
interactions inside the nucleus. The generated event conserves momentum. 

A similar two-step procedure is used to generate the J/ip — > sample, 
differing only in the method for achieving agreement of J/ip kinematic distri- 
butions with previous measurements. In this case, the generated events are re- 
weighted according to the parameterizations of the J/ip differential cross sec- 



and dajdxp oc (1 - |x F |) n , with B = 3.00 ± 0.02 GeV/c and n = 4.91 ± 0.18. 

The detector response is simulated with the GEANT 3.21 package [24]. Re- 
alistic detector efficiencies, readout noise and dead channels are taken into 
account. The simulated events are processed by the same trigger and recon- 
struction codes as the data. We have checked that the mis- identification prob- 
abilities of pions from K s decay, kaons from 0(1020) decay and protons from 
A decay estimated by the Monte Carlo agree with those measured in the data. 



4 The Data Sample and Analysis 

4-1 Introduction 

During data-taking, the interaction rate was maintained at approximately 

5 MHz, resulting in an average of approximately 0.6 interactions per filled 
HERA bunch. A total of 50 x 10 6 triggers were recorded, with 38% from runs 
with one or two carbon targets, 55% from runs with carbon and tungsten 
targets operated simultaneously, 5% from runs with one titanium wire and 
2% from runs with a single tungsten wire. After eliminating runs with prob- 
lematic detector performance, poor beam conditions or non-standard trigger 



tions in proton-gold collisions at 800 GeV by E789 [23] : da jdp\ oc 
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conditions, 47 x 10 6 events remain. 

The events are reconstructed with the standard HERA-B analysis package 
[25,26]. The search for J ftp and D° candidates is performed by analyzing the 
mass spectrum for unlike-sign dimuon candidates. D° candidates are further 
selected by requiring a detached vertex. The number of background events in 
the signal region is calculated from sidebands. The upper limit is obtained by 
comparing the observed number of events inside the signal region with the 
background estimate and normalizing to the number of observed J/V's, after 
appropriate corrections are applied. 

The mass spectrum for unlike-sign dimuon candidates after minimal cuts is 
shown in Fig. 2. Peaks at the u/p, 0, J/ip and i/)(2S) masses are clearly vis- 
ible. The fall-off towards low mass is caused by an implicit pr cut of about 
0.7GeV/c imposed by the SLT when defining regions of interest for subse- 
quent tracking. A fit in the mass interval 2.4-3.5 GeV/c 2 to an exponential 
background plus a Gaussian with a radiative tail yields 147710 ± 520 J/ijj de- 
cays in a two standard deviation window centered on the mean value from the 
Gaussian fit: 3.095 GeV/c 2 . The fitted width is 44MeV/c 2 . The mean value 
is 1.9MeV/c 2 below the PDG value [8]. From these numbers we estimate the 
mass resolution in the D° region to be about 25MeV/c 2 and that the re- 
constructed mass of a D° — > p + p~ signal would be within l.lMeV/c 2 of the 
PDG [8] value. 



4-2 Event Selection 

The main background for the D° — > p + p~ channel results from muon pairs 
from independent ^ or K ± decays which appear to form a secondary vertex 
displaced from the primary vertex. The cuts are designed to minimize this 
background while maintaining high efficiency for D° decays. 

The data sample is first reduced by application of relatively loose cuts. In 
this first pass, a general vertex-finding algorithm is applied and events with at 
least one reconstructed primary vertex and at most one reconstructed primary 
vertex per target wire are selected. Muon candidate tracks are then selected 
by requiring that the muon probability, V^, derived from the MUON hit infor- 
mation, be greater than 0.01, that the \ 2 P er degree of freedom of the track fit 
(x 2 r /d.o.f.) be less than 20, and that the kaon identification likelihood proba- 
bility from the RICH reconstruction be less than 0.4. The two muon tracks are 
then excluded from the primary vertex, and primary and dimuon vertices are 
fitted. The \ 2 probability of the primary vertex is required to be greater than 
0.01 and that of the dimuon vertex greater than 0.2. The number of tracks 
per primary vertex is required to be less than 50. Approximately 0.93 million 
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dimuons with mass in the D° region and 69000 J/ip decays (estimated here 
and below using the fit described in Sec. 4.1) survive the initial cuts. 

In the following, we call the muon pair with a fitted secondary vertex a 
"dimuon pseudoparticle" . We consider two regions of invariant mass: 

2.7-4.0 GeV/c 2 , which we refer to as the "J/ip region", and 1.59-2.15 GeV/c 2 , 
which we refer to as the "D° region" . 

Further selection cuts are divided into two parts: 

• Common cuts applied both for D° and J/ip regions. These cuts are mainly 
quality cuts, the efficiencies of which are nearly identical for muons from 
possible D° — > fi + fi~ and for muons from J/ip — > fi + fi~. Small differences in 
the efficiencies coming from the different momentum distributions of muons 
from D° decay and J/ip decay are evaluated from the data and Monte Carlo. 

• Cuts applied only for the D° region (lifetime cuts). These cuts are intended 
to select a well-defined detached secondary vertex associated with the se- 
lected primary vertex. 

The following common selection criteria are applied for D° and for J/tp regions: 

• a cut on total track multiplicity (N tr ) to suppress multi-event pile-up which 
is enhanced by the two- muon requirement; 

• a cut on Xtr/d-o.f. for each muon to suppress ghosts and 7r/K decays in 
flight; 

• a cut on Vfj, to reduce fake dimuon events; 

• a cut on the transverse momentum of each muon {pj) to suppress muons 
from pion and kaon decays. 

To optimize the common cuts, we employ a blind analysis technique: all 
dimuons from the D° signal region are masked and the cuts are chosen to 
maximize the quantity Nj/^/y/ £> D o , where Nj/^ is the number of J/tp candi- 
dates above background found in the invariant mass spectrum in a two 
standard deviation window around the J/ip position and B D o is the expected 
background in the D° signal region (1.815 — 1.915 GeV/c 2 ), estimated from 
D° sidebands (1.59 - 1.79GeV/c 2 and 1.94 - 2.14GeV/c 2 ). The resulting 
cuts are: N tr < 45, xl/d.o.f. < 7.5, > 0.7, f T > 0.7GeV/c. 

After all common cuts are applied, about 238000 events in the D° region and 
46000 events in the J/ip peak remain. The dimuon mass distributions for D° 
and J/ip regions with the above cuts are shown in Fig. 3. 
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4-3 D° Selection 



To isolate possible D° mesons, cuts are applied to three quantities: the sepa- 
ration between primary and secondary vertices, the proper decay length, and 
the impact parameter of the dimuon pseudoparticles to the primary vertex. 
At HERA-B energies, nearly all D° mesons originate at the target wires, i.e., 
the fraction arising from B decays is negligible (< 0.1%). 

Most D° mesons decay within a few millimeters of the primary vertex in the 
laboratory frame. This distance is comparable to the precision of the secondary 
vertex measurement in the longitudinal direction. To ensure that the secondary 
vertex is well separated from the associated primary vertex, we compare this 
distance with uncertainties of primary and secondary vertices and cut on the 
separation value Az = (z sec — z pr .) / <Jo^ c + a Zpr , where z pr and z sec are the 
^-coordinate (along the beam direction) of primary and secondary vertices, 
respectively, and cr Zsec , cr Zpr are their calculated errors. The average resolutions 
are a z 3 ec — 500 jum and a Zpr = 420 jum. 

The proper decay length is given by cr = mc ■ L/p, where m is the dimuon 
invariant mass, L is the decay length in the laboratory frame, and p is the 
reconstructed momentum of the dimuon pseudoparticle. 

The impact parameter (/„) is defined as the distance between the primary 
vertex and the point of intersection of the dimuon pseudoparticle flight direc- 
tion with the xy plane at the z position of the primary vertex. The impact 
parameter resolution is typically in the range of 30 jum to 60 jum. 

To optimize the cuts, we apply a blind analysis technique similar to that used 
to optimize the common cuts. Since these three quantities are correlated, a 
three-dimensional optimization is performed wherein the ratio N^ C / S is max- 
imized. N^ c is the number of reconstructed Monte Carlo events in the D° 
peak. The "experimental sensitivity", S*, is the average 90% confidence level 
upper limit on the number of signal events obtained for an ensemble of ex- 
periments with the expected background estimated from the D° sidebands, 
assuming no signal from D° — > fi + fi~ is present (from Table XII of Ref. [27]). 

This optimization procedure produces the following cuts: Az > 7.0, cr > 
250 /im and I v < 110 /zm. The optimized experimental sensitivity is S opt = 5.53 
events. 
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5 Results 



After applying all cuts, 31 events remain in the D° mass region as shown 
in Fig. 4. The sidebands are indicated by dashed lines and signal region is 
indicated by dotted lines. The signal region contains 3 events. 

The number of background events in the D° mass region from D° — > K~7r + 
decays is estimated from Monte Carlo to be 1.8 ± 1.0. All other D° decay 
modes give a negligible contribution. The background shape is therefore not 
significantly influenced by charm decay and we estimate the background using 
the shape of the mass plot before D° selection cuts are applied (see Fig. 3a). 
After normalizing to the number of events in the sidebands, the expected 
background in the signal region is 6.0 ± 1.2. We have checked that the back- 
ground shape does not significantly change relative to the uncut distribution 
when one of the three selection cuts is removed (three tests). The maximum 
difference between background estimates derived from the uncut distribution 
and the three partially cut distributions is 4%. We note that a simple linear 
interpolation between sidebands also predicts 6.0 ±1.2 background events in 
the signal region. 

The upper limit calculation for the D° — > decay mode requires knowl- 

edge of the D° to J/ip production cross section ratio in proton-nucleus inter- 
actions. Three experiments have published results on the D° production cross 
section at 800 GeV. The x F coverages of experiments E653 [20] and E743 [21] 
are similar to that of HERA-B: —0.2 < xp < 0.1. The xp coverage of exper- 
iment E789 [28] is relatively small and the E653 and E743 parameterizations 
have been used to extrapolate to the full xf range. For this reason we do not 
include it in the calculation of the D° cross section. From the remaining two 
measurements of 22ty ± 5 /ib/nucleon and 38 ± 3 ± 13 /ib/nucleon we obtain 
o$ = 27.3 ± 7.7 /ib/nucleon. 

The prompt J/ip production cross section per nucleon, 0jy^, has been measured 
by two fixed target experiments [23,29] at 800 GeV. Both measurements were 
performed with nuclear targets and extrapolated to atomic weight (A) of 1, 
assuming <7j^ = Uj^, • A aj /^. After adjusting these measurements using 
the most recent measurement of aj/f = 0.955 ± 0.005 [30], and averaging 
them we obtain a prompt J/ip production cross section of <jj^, = 333 ± 6 ± 
26nb/nucleon at 800 GeV. We assume that the ratio of D° and J/ip cross 
sections does not change significantly between 800 GeV and 920 GeV. 

The ratio of acceptances, not including cuts applied only for selecting D° — > 
is a D o/aj/^ = 0.287 ± 0.028. The error is evaluated by studying the efficiencies 
of the applied cuts when detector characteristics (channel efficiencies, mask- 
ings, resolutions) are varied within ranges that reflect their uncertainties. The 



11 



quadratic sum of contributions from all significant sources, including trigger- 
ing, target geometry, muon and kaon identification and Monte Carlo statistics 
is 9.8%. The efficiency of the additional secondary vertex cuts applied in the 
D° region is e D o = (6.83 ± 1.08) x 10~ 2 . 

Data samples from the various targets are combined into a single data set 
assuming the production cross sections per nucleon depend on atomic weight 
(A) as o$ {JM = Oi$ M yA aDO wi>), with a B o = 1.02 ±0.03 ± 0.02 (E789) [28] 
and a 3/t = 0.955 ± 0.005 (E866) [30]. 

Equation (1) can be rewritten as: 

£(D° -+ < ^ (2) 



where 



a D o 




e D o 









pN 



< B(J/V> 



(3) 



is the detector sensitivity after summation over targets. 

The values and associated errors for all terms in equation (3) are given in 
Table 1. Using these numbers, F sens = 1.57 x 10 6 . After combining statistical 
and systematic errors quadratically the total relative error on F sens from all 
contributing terms in equation (3) is 37%. 

Table 1 

Values and contributions to the systematic uncertainty on B(D° — ► fi + [i~) for the 
factors in equation (3). (* - statistical and systematic errors added quadratically) 



factor 


value 


% 


°D° / a 3/il> 
e D o 

<7j^ (nb / nucleon) 
Ojyo {fih 1 nucleon) 
B(J/^ - M + aO 


0.287 ± 0.028 
(6.83 ±1.08) x 10~ 2 
333 ± 6 ± 26 
27.3 ± 7.7* 
(5.88 ±0.10) x 10~ 2 
31010 ± 200(stat) 
12660 ± UO(stat) 
2430 ± 60(stat) 
0.065 ± 0.036* 


9.8 
15.8 
8.0 
28.2 
1.7 
0.7 
1.1 
2.5 
12.3 
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Recent publications [31] employ a variety of methods for calculating upper 
limits and there is no universally accepted procedure [27,32,33]. We choose 
an approach similar to that first advocated by Feldman and Cousins [27]. 
This method has been since extended by Conrad et al. [34] to incorporate 
uncertainties in detector sensitivity and the background estimate based on an 
approach described by Cousins and Highland [35] . A further refinement of the 
Conrad et al. method by Hill [36] results in more appropriate behavior of the 
upper limit when the observed number of events is less than the estimated 
background, as is the case for the present measurement. We have adopted this 
method but note that Table 2 contains all of the numbers needed to calculate 
an upper limit using any of the methods in the papers cited above. We assume 
that the probability density functions of F sens and background estimates are 
Gaussian-distributed. 

Using equation (2) and taking systematic errors and background fluctuations 
into account with the Hill prescription [37], the upper limit at 90% confidence 
level on B(D° -> fi+fj,-) is 2.0 x 10~ 6 . 

Table 2 

Summary of parameters entering into the upper limit calculation 



Sensitivity factor, F sens 


(1.57 ± 0.58) x 10 6 


Number of events in the signal region 


3 


Expected background events 


6.0 ± 1.2 



6 Conclusions 

We have investigated the dimuon mass spectrum in a search for the flavor- 
changing neutral current decay D° — > Using the values of D° and J/ip 
production cross sections published in the literature we have set an upper limit 
on the branching fraction of 

B(D° -> fj+fj,-) < 2.0 x 10~ 6 

at 90% confidence level. For comparison, the lowest previously published limit 
on £(D° -> fi+fi-) is 2.5 x 10~ 6 at 90% confidence level [5]. 

According to the recent calculations by Burdman et al. [4], in the case of strong 
i?-parity violation in the MSSM the D° — > fi + fi~ branching fraction is at the 
level of 3.5 x 10 -6 . Thus, our result disfavors this scenario and constrains the 
size of .R-parity violating couplings which enter into the predicted branching 
fraction. 
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Fig. 2. Invariant mass of unlike-sign dimuon candidates after minimal cuts. The 
positions of known resonances are indicated. 
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Fig. 3. Candidates with two oppositely charged muon tracks for (a) D° and (b) J/tp 
regions after all common (see text) cuts have been applied. The curve in (b) is the 
result of a fit to an exponential to describe the background and a Gaussian with a 
radiative tail to describe the signal. The two standard deviation selection window 
around the J/ip position is indicated by the dashed lines. 
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Fig. 4. Unlike-sign dimuon mass spectra in the D region after all cuts have been 
applied. The sidebands are indicated by dashed lines and the signal region is in- 
dicated by dotted lines. The numbers of events in each of the regions are given. 
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